docs: draft MySQL snapshot parallelism page - #38091
Conversation
4d26654 to
c530367
Compare
7cd15a2 to
6fb412a
Compare
6fb412a to
ad4508e
Compare
ad4508e to
bedaa57
Compare
bedaa57 to
07af68f
Compare
a0bd0d7 to
1d5f164
Compare
1d5f164 to
4aa343a
Compare
4aa343a to
a8438ee
Compare
a8438ee to
529b0be
Compare
529b0be to
bdce9de
Compare
bdce9de to
b7a0bc2
Compare
d189c40 to
132f2b4
Compare
132f2b4 to
1bdd206
Compare
1bdd206 to
9329cfa
Compare
9329cfa to
1d8fe51
Compare
1d8fe51 to
891ad5f
Compare
891ad5f to
2167caa
Compare
2167caa to
1805742
Compare
1805742 to
9efe79a
Compare
9efe79a to
b105665
Compare
b105665 to
7d24859
Compare
7d24859 to
bc6f2ff
Compare
bc6f2ff to
28fd467
Compare
User-facing draft for the parallel snapshot feature: how key ranges divide across workers, single-column string PK requirement and fallback, upstream connection and statistics considerations, and where to observe progress. Marked private preview.
…ance Make the MySQL page strictly MySQL-specific (split eligibility, prefix-based partitioning, connection counts) and move the cross-source worker-scaling discussion into a shared snapshotting parallelism include on the snapshotting concept page and the ingest overview. Add a troubleshooting section on telling whether the upstream database is the bottleneck (CPU, IOPS, throughput, connections, cache, replica lag) and what to do about it. Present the feature as production, without a preview banner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018yzDA9ywnCLXweNCzqBm12
| headless: true | ||
| --- | ||
|
|
||
| Materialize parallelizes snapshotting across the workers of the cluster |
There was a problem hiding this comment.
Since we don't parallelize everything (sql server), I would make it more "can" and separate out. I think something like (?):
Materialize can parallelize snapshotting across the workers of the cluster
hosting the source.
- **PostgreSQL sources** are parallelized by table, i.e., different tables are
read concurrently by different workers. On PostgreSQL 14 and later,
Materialize additionally attempts to partition each table's read across
workers; tables that cannot be partitioned fall back to a single worker.
- **MySQL sources** are parallelized by table, i.e., different tables are read
concurrently by different workers. For tables that meet [certain
requirements](/ingest-data/mysql/snapshot-parallelism/), Materialize
additionally partitions the table's read across workers. See [MySQL snapshot
parallelism](/ingest-data/mysql/snapshot-parallelism/).
- **Kafka sources** are parallelized by topic partition, with partitions distributed
across workers. Parallelism is bounded by the topic's partition count.
- **SQL Server sources** are not parallelized; a single worker reads all tables.
I might have misinterpreted something ... but, hopefully, the above can give a gist of what I'm thinking
There was a problem hiding this comment.
Adopted your structure, thanks. Only tweak: split the PG bullet's fallback clause into its own sentence.
| so parallelism is bounded by the topic's partition count. SQL Server sources | ||
| are not parallelized: a single worker reads all tables. | ||
|
|
||
| A cluster's [size](/sql/create-cluster/#available-sizes) determines its |
There was a problem hiding this comment.
?
The degree of snapshot parallelism depends on the number of workers. A cluster's
[size](/sql/create-cluster/#available-sizes) determines its number of workers,
so a larger cluster shortens the snapshot. The volume read from the upstream
database is unchanged; it is compressed into a shorter window of more concurrent
queries and connections. To determine whether snapshotting is overloading the
upstream database, and for ways to mitigate the load, see [Is the upstream
database
overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded).
|
|
||
| - **Read load.** Snapshotting puts read, CPU, and network load on the upstream | ||
| system, proportional to the data volume. | ||
| system, proportional to the data volume and concentrated in proportion to |
There was a problem hiding this comment.
? Is it more?
- **Read load.** Snapshotting puts read, CPU, and network load on the upstream
system. The total load is proportional to the volume of data being
snapshotted, while the [source cluster's parallelism](#parallelism) affects
the peak load: more workers compress the reads into a shorter window.
?
There was a problem hiding this comment.
Adopted the total-vs-peak framing.
| - **Network throughput** at the instance type's cap. | ||
| - **Memory** pressure, or a falling cache hit rate as large scans evict the | ||
| normal workload's working set. | ||
| - **Connections** near the database's limit. Snapshotting opens connections |
There was a problem hiding this comment.
The snapshotting sentence isn't true/applicable for kafka and sql server, yes?
There was a problem hiding this comment.
Right. Scoped the bullet to PostgreSQL and MySQL sources.
| ## Which tables are split | ||
|
|
||
| The snapshot of an individual table is split across workers when all of the | ||
| following hold: |
There was a problem hiding this comment.
?
Materialize splits the snapshot of an individual table across workers when all of the following conditions are met:
| values. See [How a table is partitioned](#how-a-table-is-partitioned). | ||
|
|
||
| Tables that don't meet these requirements, or whose boundary sampling fails | ||
| for any reason, still snapshot correctly: each is read in full by a single |
There was a problem hiding this comment.
Could we simplify and add a link to the boundary sampling?
If a table does not meet these requirements, or if the boundary sampling fails,
its snapshot is not split: a single worker reads the table in full.
There was a problem hiding this comment.
Adopted, with one extra sentence noting that different tables are still read concurrently.
|
|
||
| ## How a table is partitioned | ||
|
|
||
| Materialize partitions a table by the unique leading characters of its |
There was a problem hiding this comment.
an eligible table and maybe separate with a little tweak?
Materialize partitions an eligible table using the
leading characters of its primary key values. Before reading the table,
Materialize probes the primary key index to discover key prefixes and uses the
MySQL optimizer's row estimates to gauge how many rows fall under each prefix.
It extends the prefixes as needed to find boundaries that divide the table into
roughly even ranges. The probes are inexpensive point lookups, capped in
proportion to the table's estimated size, so the sampling phase stays negligible
next to the snapshot itself.
Each worker then reads only its assigned range, within the same
consistent snapshot of the upstream database, so the result is identical to a
single-worker snapshot, only faster.
There was a problem hiding this comment.
Adopted your two-paragraph version.
|
|
||
| - **Connection count.** While the snapshot is being set up, Materialize | ||
| briefly holds up to two connections per worker, plus one. Once reading is | ||
| underway, this settles to one connection per worker reading a range, plus | ||
| one coordination connection. After the snapshot completes, the source drops | ||
| back to a single replication connection. If your MySQL server or connection | ||
| pooler enforces a low | ||
| [`max_connections`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections) | ||
| limit, account for this burst when sizing it. | ||
|
|
||
| - **Statistics freshness.** Range boundaries are placed using the MySQL | ||
| optimizer's row estimates. Stale statistics don't affect correctness, but | ||
| can skew how evenly work divides across workers. Running | ||
| [`ANALYZE TABLE`](https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html) | ||
| on very large tables before creating the source can improve balance. |
There was a problem hiding this comment.
There was a problem hiding this comment.
I'd keep these here for now: both bullets only apply when parallel snapshotting is active, which is private preview and flag-off, so they'd be noise in the general MySQL considerations. Worth revisiting when the feature is on by default.
Mark the table-splitting feature as private preview since it ships flag-off, restructure the shared parallelism include as a per-source list and caveat that a larger cluster can shorten the snapshot only as far as the work parallelizes, scope the troubleshooting connection guidance per source type, rework the overload mitigations around upsizing the database or dropping the source and retrying, point observability at the existing monitoring page, and add discovery blurbs to the MySQL and PostgreSQL ingest overview pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgSuGEvoVT9TFHAgq6FCi
Part of SS-358.
Adding docs for the new MySQL parallel snapshotting feature, some more context on the load we place on upstream databases, and some high-level debugging tips for triaging obvious database issues.
The table-splitting feature ships with
mysql_source_snapshot_parallelismoff by default, so it is documented as private preview: a banner on the MySQL snapshot parallelism page and inline preview markers where the splitting is mentioned elsewhere.🤖 Generated with Claude Code
https://claude.ai/code/session_016AgSuGEvoVT9TFHAgq6FCi